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Abstract 

Background: The quality of reporting in systematic reviews (SRs)/meta-analyses (MAs) of diagnostic tests published by 
authors in China has not been evaluated. The aims of present study are to evaluate the quality of reporting in diagnostic 
SRs/MAs using the PRISMA statement and determine the changes in the quality of reporting over time. 

Methods: According to the inclusion and exclusion criteria, we searched five databases including Chinese Biomedical 
Literature Database, PubMed, EMBASE, the Cochrane Library, and Web of knowledge, to identify SRs/MAs on diagnostic 
tests. The searches were conducted on July 14, 2012 and the cut off for inclusion of the SRs/MAs was December 31 st 201 1. 
The PRISMA statement was used to assess the quality of reporting. Analysis was performed using Excel 2003, RevMan 5. 

Results: A total of 312 studies were included. Fifteen diseases systems were covered. According to the PRISMA checklist, 
there had been serious reporting flaws in following items: structured summary (item 2, 22.4%), objectives (item 4, 18.9%), 
protocol and registration (item 5, 2.6%), risk of bias across studies (item 15, 26.3%), funding (item 27, 28.8%). The subgroup 
analysis showed that there had been some statistically significant improvement in total compliance for 9 PRISMA items after 
the PRISMA was released, 6 items were statistically improved regarding funded articles, 3 items were statistically improved 
for CSCD articles, and there was a statistically significant increase in the proportion of reviews reporting on 22 items for SCI 
articles (P<0.050). 

Conclusion: The numbers of diagnostic SRs/MAs is increasing annually. The quality of reporting has measurably been 
improved over the previous years. Unfortunately, there are still many deficiencies in the reporting including protocol and 
registration, search, risk of bias across studies, and funding. Future Chinese reviewers should address issues on these 
aspects. 
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Introduction 

Systematic reviews (SRs) and meta-analyses (MAs) of diagnostic 
tests will potentially have an increasing role in healthcare as 
decision makers check the evidence before implementing new 
diagnostic technologies [1]. However, only high quality diagnostic 
SRs can provide the best evidence for clinical decision makers. 
Low quality SRs may mislead clinical practice, which may arise at 
a primary study level, due to flaws in the design, execution and 
reporting of the component studies [2,3]. Like all other types of 
SRs, diagnostic SRs are also prone to a number of shortcomings 
[!]• 

In order to improve the quality of diagnostic SRs, quality 
evaluation of the primary studies has become an important 
element of the review process. The publication of the Quality 
Assessment of Diagnostic Accuracy Studies (QUADAS) tool [4] 



provides investigators with a means of assessing the major domains 
that affect the validity of diagnostic study. As a matter of fact, some 
experiences, reports, feedbacks from users or academic organiza- 
tions have suggested the potential to improve this tool. Therefore, 
the QUADAS Group revised it and then launched QUADAS-2 
[5] in 2011 [6]. The Standards for the Reporting of Diagnostic 
accuracy studies (STARD) initiative [3] has given investigators a 
reporting framework that undoubtedly improves the quality of 
primary studies. 

SRs of diagnostic tests have been developing for nearly two 
decades abroad [1]. The first SR of diagnostic tests was published 
in Chinese journal in 2001 [7]. Since then, an increasing numbers 
of SRs of diagnostic tests have been published in China, however, 
the methodological and reporting quality of these reviews varies 
widely. Therefore, it is vital to assess the quality of diagnostic SRs 
before being used for healthcare policy or clinical decision making. 
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Table 1. Data extraction items of included studies. 







Items 


Interpretation 


Basic information 


1. Publication year 


Year of publication of SRs/Mas 




2. Source 


Journal or degree paper 




3. Language of publication 


Chinese or English 




4. Number of authors 


No. authors of writing SR/MAs 




5. Departement of authors 


Number and role of authors' departement 




6. Gold standard 


Reference standard on diagnostic tests 




7. Index test 


Evaluated tests 




8. Information of included RCT 


Language and number of included RCT 




9. Title 


Identify the report as a SR, MA, or other 




10. Foundation item 


Number and nature on foundation 




1 1 . Categories of disease 


Condition focused on in review (ICD-10) 




12. Quality assessment on RCT 


Information of quality assessment tool on RCT 


Quality assessment of reporting 
information 


13. Title 




14. Abstract 


15. Introduction 




16. Methods 


Yes/Partial/No for per item 


17. Results 


18. Discussion 


19. Funding 



(Notes: RCT: Randomized Controlled Trials; SR: Systematic review; MA: Meta-analysis). 
doi:1 0.1 371 /journal.pone.0085908.t001 



3 214 citations retrieved from search 
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Excluded after screening full text (n=43): 
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► 

t 



Included studies (n=3 12) 



Figure 1. Flow chart of article screening and selection process. 

doi:1 0.1 371 /journal.pone.0085908.g001 
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Figure 2. Number of included systematic reviews/Meta-analyses per year of publication. 

doi:1 0.1 371 /journal.pone.0085908.g002 



Methodological quality considers how well the SR is conducted 
(including literature searching, pooling of data, etc.). Reporting 
quality considers how well systematic reviewers have reported 
their methodology and findings [8] . There is no specialized tool to 
evaluate the quality of SRs of diagnostic research. As a newer 
standard of reporting SR, the Preferred Reporting Items for 
Systematic Reviews and Meta-Analyses (PRISMA) was released to 
replace the Quality of Reporting of Meta-analyses (QUOROM) 
for guiding the review reporting [9]. The updated PRISMA 
statement was based on the conceptual and practical advances that 
had been made in the science of SRs. The complete PRISMA 
statement consists of a 27-item checklist, along with a flow diagram 
[10]. Most of the checklist items are relevant when reporting SRs 
of non-randomized studies assessing the benefits and drawbacks of 
interventions, however, the need to modify or incorporate 
additional items for diagnostic accuracy studies should be 
recognized [11]. 

Willis BH et al. have investigated the quality of reporting of 
MAs in diagnostic researches published in English prior to 
December 31 st , 2008, and found that the quality of reporting 
had many flaws [1]. It is unclear whether the quality of reporting 
of SRs/MAs on diagnostic researches published by authors in 
China have a similar result, and further validation is necessary. 
Thus, the objective of this study is to examine the quality of 
reporting of published SRs/MAs in diagnostic tests by authors in 
China, according to their compliance with the PRISMA 
statement. 

Methods 

Data Sources and Searches 

Two independent reviewers systematically searched the follow- 
ing electronic databases: Chinese Biomedical Literature Database, 
PubMed, EMBASE, the Cochrane Library, and Web of science. 
Searches were conducted using a combination of the following 
terms: "systematic reviews", "systematic review", "meta analysis", 
"meta analyses", "meta-analysis", "meta-analyses", "sensitivity", 
"specificity", "China", "Chinese". The searches were implement- 
ed on July 14, 2012 and the cut off for inclusion of the SRs/MAs 
was December 31 st 2011. The syntaxes were adjusted correspond- 
ing to different database. The detailed search algorithms for each 
database were listed in Text S 1 . 



Selection 

Two independent reviewers selected articles according to the 
inclusion and exclusion criteria designed in advance. Disagree- 
ments were resolved in consultation with Dr. Tian. After 
discarding duplication studies by Endnote X3 (The Thomson 
Reuters, Britain), we reviewed all the abstracts, identified 
potentially eligible articles and citations for which a decision 
could not be made from the abstract. We then managed to retrieve 
the full - text of these articles to determine whether they were 
eligible. 

The inclusion criteria were as follows: (a) The SRs/MAs on 
diagnostic tests were published by authors in China. The 
diagnostic test was defined as a technology to distinguish between 
patients with disease (or more generally, a specified target disorder) 
and those without disease. Target disorders were considered to be 
pathological processes and not related to a success or failure of an 
intervention, such as successful placing of stents. In such a test 
accuracy study, the results of 'index test' were compared with 
those of the reference standard determined in the same patient. 
The reference standard should be the best available method for 
identifying patients that have the target disorders, (b) One of the 
objectives of the included studies was to estimate a measure of the 
performance of the diagnostic test; articles that reported relevant 
data would be extracted; (c) The qualitative and quantitative 
comprehensive analyses were performed for all included studies; 
(d) The search terms should be explicitly stated and include 
sensitivity or specificity. 

The exclusion criteria were as follows: (a) Repeated literature, 
review and methodological literature, original research, animal 
research and case reports, (b) Conference abstracts and letters to 
the journal editors; (c) The SRs/MAs on other fields. 

Quality Assessment 

The PRISMA statement [10] which consists of a twenty seven- 
point checklist was used to evaluate the overall quality of reporting 
of meta analyses. To indicate the degree of compliance, each 
checklist item was assigned one of three responses: 'Yes' for total 
compliance; 'partial' for partial compliance; and 'No' for non- 
compliance. 

Data Collection and Analysis 

Data collection was carried out independently by two reviewers 
using standard data extraction forms (Table 1). All disagreements 
were resolved by discussion or the third researcher. 
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Table 2. Characteristics of included systematic reviews/meta-analyses. 







Items 


n(%) 


95%CI 


Language of publications 


Chinese 


235(75.3) 


(70.2-79.8) 




English 


77(24.7) 


(20.2-29.8) 


Category of journals 


CSCD 


134(57.0) 


(50.6-63.2) 




Non-CSCD 


101(43.0) 


(36.8-49.4) 




SCI 


74(23.7) 


(19.3-28.8) 




Non-SCI 


238(76.3) 


(71.2-80.7) 


Number of authors 


<2 


37(11.9) 


(8.7-15.9) 




>3 


275(88.1) 


(84.1-91.3) 


Affiliations 


University 


105(33.7) 


(28.6-39.1) 




Hospital 


207(66.3) 


(60.9-71.4) 


Number of affiliations 


1 


162(51.9) 


(46.4-57.4) 




>2 


150(48.1) 


(42.6-53.6) 


Foundation item 


Government 


90(28.8) 


(24.1-34.1) 




Unreported 


222(71.2) 


(65.9-75.9) 


Index test 


Imaging technologies 


141(45.2) 


(39.8-50.8) 




Laboratory technologies 


159(51.0) 


(45.4-56.5) 




Pathological technologies 


2(0.6) 


(0.2-2.5) 




Others 


10(3.2) 


(1.7-5.9) 


Number of included RCT 


1-5 


21(6.7) 


(4.4-10.1) 




6-10 


88(28.2) 


(23.5-33.5) 




11-15 


80(25.6) 


(21.1-30.8) 




16-20 


43(13.8) 


(14-18.1) 




>21 


78(25.0) 


(20.5-30.1) 




Unreported 


2(0.6) 


(0.2-2.5) 


Categories of disease 


Neoplasms 


132(42.3) 


(36.9-47.9) 




Diseases of the digestive system 


32(10.3) 


(7.3-14.1) 




Diseases of the respiratory system 


24(7.7) 


(5.2-11.2) 




Diseases of the circulatory system 


26(8.3) 


(5.7-12.0) 




Certain infectious and parasitic diseases 


32(10.3) 


(7.3-14.1) 




Diseases of the blood and blood -forming organs 


8(2.6) 


(1.3-5.0) 




Endocrine,nutritional and metabolic diseases 


9(2.9) 


(1.5-5.4) 




Diseases of the musculoskeletal system and connective tissue 


17(5.4) 


(3.4-8.6) 




Diseases of the eye and adnexa 


2(0.6) 


(0.2-2.5) 




Diseases of the genitourinary system 


10(3.2) 


(1 .7-5.9) 




Pregnancy, childbirth and the puerperium 


3(1.0) 


(0.3-2.9) 




Congenital malformations, deformations and 
chromosomal abnormalities 


7(2.2) 


(1.1-4.6) 




Symptoms, signs and abnormal clinical and laboratory findings, 
not elsewhere classified 


2(0.6) 


(0.2-2.5) 




Injury, poisoning and certain other consequences of external causes 


5(1.6) 


(0.7-3.8) 




Certain conditions originating in the perinatal period 


3(1.0) 


(0.3-2.9) 



(Notes: RCT: Randomized Controlled Trials; CSCD: Chinese Science Citation Database; SCI: Science Citation Index). 
doi:1 0.1 371 /journal.pone.0085908.t002 



Extraction of data included the following items: title; publication 
year; publication journals; publication language; number of 
authors; affiliations of authors; number of affiliations; index test; 
reference standard; funding sources; categories of disease; method 
used to assess the quality for original studies including QUAD AS, 
and responses to the PRISMA statement. 

According to An N et al. [12], we performed subgroup analysis 
on the quality of reporting for total compliance in each items of 



PRISMA by t-test. The factors of subgroup analysis were 
presented as following: the year of publication (<2008 vs. 
^2009), the affiliations (hospital vs. university), funding sources 
(funded vs. non-funded), Chinese Science Citation Database 
(CSCD) vs. non-CSCD, Science Citation Index (SCI) vs. non- 
SCI. The odds ratio (OR) value and 95% CI was used as the 
summary statistic for subgroup comparisons through Mantel- 
Haenszel and Std, by using a fixed-effect model. If the OR was 
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Figure 3. Changing pattern of quality assessment in diagnostic systematic reviews/Meta-analyses. 

doi:10.1371/journal.pone.0085908.g003 

undefined, then Fisher's exact test was used. The ORs for each better than group 2. Statistical significance was set at P^O.050. 
item represent responses of "Yes" for group 1 versus responses of Analysis was performed using Excel 2003 and RevMan 5.0 
"Yes" for group 2; OR>l represents that the quality of group 1 is software. 



Table 3. The results of quality assessment of reporting (n = 312). 







PRISMA Items 


Yes 




Partial 




No 








n(%) 


95%CI 


n(%) 


95%CI 


n(%) 


95%CI 


Title 


1 .Title 


257(82.4) 


77.7-86.2 


0 




55(17.6) 


13.8-22.3 


Abstract 


2. Structured summary 


70(22.4) 


18.1-27.4 


228(73.1) 


67.9-77.7 


14(4.5) 


2.7-7.4 


Introduction 


3-Rational 


280(89.7) 


85.9-92.7 


0 




32(10.3) 


7.1-14.1 




4-Objective 


59(18.9) 


14.9-23.6 


247(79.2) 


74.3-83.3 


6(1.9) 


0.9-4.2 


Methods 


5. Protocol and registration 


8(2.6) 


1.3-5.0 


0 




304(97.4) 


95.0-98.7 




6. Eligibility criteria 


275(88.1) 


84.1-91.3 


19(6.1) 


3.9-9.3 


16(5.1) 


3.2-8.2 




7. Information sources 


274(87.8) 


83.7-91.0 


0 




38(12.2) 


9.0-16.3 




8.Search 


120(38.5) 


33.2-44.0 


129(41.3) 


36.0-46.9 


63(20.2) 


16.1-25.0 




9. Study selection 


1 34(42.9) 


37.6-48.5 


62(19.9) 


15.8-24.7 


116(37.2) 


32.0-42.7 




10. Data collection process 


193(61.9) 


56.3-67.1 


40(12.8) 


9.5-17.0 


79(25.3) 


20.8-30.4 




11. Data items 


148(47.4) 


41.9-53.0 


51(16.3) 


1 2.6-20.9 


113(36.2) 


31.1-41.7 




12. Risk of bias in individual 
studies 


204(65.4) 


59.9-70.5 


13(4.2) 


2.4-7.0 


95(30.4) 


25.6-35.8 




13. Summary measures 


232(74.4) 


69.2-78.9 


0 




80(25.6) 


21.1-30.8 




14.Synthesis of results 


266(85.3) 


80.9-88.8 


28(9.0) 


6.3-12.7 


18(5.8) 


3.7-9.0 




15. Risk of bias across studies 


82(26.3) 


21.7-31.4 


4(1.3) 


0.5-3.4 


226(72.4) 


67.2-77.1 




1 6. Additional analyses 


122(39.1) 


33.8-44.6 


8(2.6) 


1.3-5.0 


185(59.3) 


53.8-64.6 


Results 


17. Study selection 


141(45.2) 


39.8-50.8 


82(26.3) 


21.7-31.4 


89(28.5) 


23.8-33.8 




18. Study characteristics 


246(78.8) 


74.0-83.0 


20(6.4) 


4.2-9.7 


46(14.7) 


11.2-19.1 




19. Risk of bias with studies 


201(64.4) 


59.0-69.5 


12(3.8) 


2.2-6.6 


99(31.7) 


26.8-37.1 




20. Results of individual 
studies 


251(80.4) 


75.7-84.5 


23(7.4) 


4.9-10.8 


38(12.2) 


9.0-16.3 




21. Synthesis of results 


235(75.3) 


70.2-79.8 


30(9.6) 


6.8-13.4 


47(15.1) 


11.5-19.5 




22. Risk of bias across studies 


98(31.4) 


26.5-36.8 


3(1.0) 


0.3-2.9 


21 1 (67.6) 


62.2-72.6 




23. Additional analyses 


131(42.0) 


36.6-47.5 


7(2.2) 


1.1-4.6 


1 74(55.8) 


50.2-61.2 


Discussion 


24. Summary of evidence 


267(85.6) 


81.2-89.1 


37(11.9) 


8.7-15.9 


8(2.6) 


1.3-5.0 




25. Limitations 


212(67.9) 


62.6-72.9 


40(12.8) 


9.5-17.0 


60(19.2) 


15.2-24.0 




26. Conclusions 


155(49.7) 


44.2-55.2 


107(34.3) 


29.2-29.7 


50(16.0) 


1 2.4-20.5 


Funding 


27. Funding 


90(28.8) 


24.1-34.1 


0 




222(71.2) 


65.9-75.9 



doi:1 0.1 371 /journal.pone.0085908.t003 
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Study or Subgroup 



^2008 
Events Total 



5=2009 
Events Total 



Odds Ratio 
M-H. Fixed. 95% CI 



Odds Ratio 
M-H. Fixed. 95% CI 



Item 1 


79 


94 


179 


218 


1.15 


[0.60, 


2.20] 


Item 2 


7 


94 


63 


218 


0.20 


[0.09, 


0.45] 


Item 3 


78 


94 


202 


218 


0.39 


[0.18, 


0.81] 


Item 4 


8 


94 


51 


218 


0.30 


[0.14, 


0.67] 


Item 5 


0 


94 


8 


218 


0.13 


[0.01, 


2.29] 


Item 6 


81 


94 


195 


218 


0.73 


[0.35, 


1.52] 


Item 7 


88 


94 


187 


218 


2.43 


[0.98, 


6.04] 


Item 8 


35 


94 


85 


218 


0.93 


[0.56, 


1.53] 


Item 9 


34 


94 


100 


218 


0.67 


[0.41, 


1.10] 


Item 10 


54 


94 


140 


218 


0.75 


[0.46, 


1.23] 


Item 1 1 


29 


94 


119 


218 


0.37 


[0.22, 


0.62] 


Item 12 


42 


94 


162 


218 


0.28 


[0.17, 


0.46] 


Item 13 


68 


94 


164 


218 


0.86 


[0.50, 


1.49] 


Item 14 


76 


94 


190 


218 


0.62 


[0.33, 


1.19] 


Item 15 


20 


94 


62 


218 


0.68 


[0.38, 


1.21] 


Item 16 


36 


94 


86 


218 


0.95 


[0.58, 


1.57] 


Item 17 


22 


94 


119 


218 


0.25 


[0.15, 


0.44] 


Item 18 


60 


94 


188 


218 


0.28 


[0.16, 


0.50] 


Item 19 


37 


94 


164 


218 


0.21 


[0.13, 


0.36] 


Item 20 


74 


94 


177 


218 


0.86 


[0.47, 


1.56] 


Item 21 


66 


94 


171 


218 


0.65 


[0.37, 


1.12] 


Item 22 


23 


94 


75 


218 


0.62 


[0.36, 


1.07] 


Item 23 


39 


94 


92 


218 


0.97 


[0.59, 


1.59] 


Item 24 


75 


94 


192 


218 


0.53 


[0.28, 


1.02] 


Item 25 


50 


94 


162 


218 


0.39 


[0.24, 


0.65] 


Item 26 


42 


94 


113 


218 


0.75 


[0.46, 


1.22] 


Item 27 


27 


94 


64 


218 


0.97 


[0.57, 


1.65] 



-+- 



0.01 



Figure 4. Comparison of the quality 
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of reporting of included systematic reviews/Meta-analyses based on the publication time. 



Results 

Search 

3,214 records were retrieved as the result of searches. 2,767 
articles were excluded due to duplication, non diagnostic tests, 
conference abstracts, animal researches, non Chinese authors, or 
not being a SR/MA. After examination of the full texts of 355 
articles, a further 43 reviews were excluded because they were 
original researches, reviews, or duplication. A total of 312 SRs and 
MAs were included, 235 in Chinese and 77 in English (Figure 1). 

Characteristics of the Included SRs/MAs (Table 2) 

312 SRs/MAs were written by authors in China and published 
in 169 different journals. Only 134 (43.0%) were published in the 
journals cited by Chinese Science Citation Database, with 74 
(23.7%) being published in Science Citation Index Journals. In 
which, the impact factor (201 1) was between 0.9 and 1 1.7, and the 
impact factor for 12 reviews was higher than 5.0, dispersed 
throughout eight journals as following: Hepatology (impact factor 
(IF): 11.7), Archives of Internal Medicine (IF: 1 1.5), Thorax (IF: 
6.8), Radiotherapy and Oncology (IF: 5.6), European Journal of 
Cancer (IF: 5.5), American Journal of Kidney Diseases (IF: 5.4), 
Intensive care medicine (IF:5.4), CHEST (IF:5.3). Figure 2 



illustrated the number of SRs/MAs per publication year in the 
included set. The first diagnostic SR was published in 2001, nearly 
89.0% (278 reviews) were published after the year 2007. 
According to ICD-10 [13], 312 studies involved 15 system 
diseases. The most common disorders were neoplasms (42.3%), 
diseases of the digestive system (10.3%), and certain infectious and 
parasitic diseases (10.3%). More than half of (66.4%) the SRs/ 
MAs were written by clinicians. A wide range of diagnostic tests 
were featured in the reviews, with laboratory technologies (51.0%) 
and imaging technologies (45.2%) being the most common 
category of tests evaluated. 28.9% (90 reviews) reported funding 
sources. 88. 1 % (275 reviews) studies were composed by more than 
three authors. None of the SRs/MAs had been updated from a 
previous review. Characteristics of the included reviews were 
shown in Table 2. 

Tools of Quality Assessment of Primary Studies 

All reviews published before 2003 reported no formal quality 
assessment of the primary studies. However, most of (69.6%) 
reviews used the QUAD AS tool for quality assessment in 2011, 
although other methods of quality assessment were still being used 
(see Figure 3). 
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Study or Subgroup 



Hospital 
Events Total 



University 
Events Total 



Odds Ratio 
M-H. Fixed. 95% CI 



Odds Ratio 
M-H. Fixed. 95% CI 



Item 1 


174 


212 


84 


100 


0.87 


[0.46, 


1.65] 


Item 2 


43 


212 


27 


100 


0.69 


[0.40, 


1.20] 


Item 3 


186 


212 


94 


100 


0.46 


[0.18, 


1.15] 


Item 4 


26 


212 


33 


100 


0.28 


[0.16, 


0.51] 


Item 5 


2 


212 


6 


100 


0.15 


[0.03, 


0.75] 


Item 6 


189 


212 


86 


100 


1.34 


[0.66, 


2.73] 


Item 7 


191 


212 


83 


100 


1.86 


[0.93, 


3.71] 


Item 8 


74 


212 


46 


100 


0.63 


[0.39, 


1.02] 


Item 9 


94 


212 


40 


100 


1.19 


[0.74, 


1.94] 


Item 10 


129 


212 


64 


100 


0.87 


[0.53, 


1.43] 


Item 1 1 


99 


212 


49 


100 


0.91 


[0.57, 


1.47] 


Item 12 


134 


212 


70 


100 


0.74 


[0.44, 


1.23] 


Item 13 


159 


212 


73 


100 


1.11 


[0.65, 


1.90] 


Item 14 


181 


212 


85 


100 


1.03 


[0.53, 


2.01] 


Item 15 


49 


212 


33 


100 


0.61 


[0.36, 


1.03] 


Item 16 


80 


212 


42 


100 


0.84 


[0.52, 


1.36] 


Item 17 


92 


212 


49 


100 


0.80 


[0.50, 


1.29] 


Item 18 


163 


212 


83 


100 


0.68 


[0.37, 


1.26] 


Item 19 


131 


212 


70 


100 


0.69 


[0.42, 


1.15] 


Item 20 


170 


212 


81 


100 


0.95 


[0.52, 


1.74] 


Item 21 


160 


212 


75 


100 


1.03 


[0.59, 


1.78] 


Item 22 


60 


212 


38 


100 


0.64 


[0.39, 


1.06] 


Item 23 


84 


212 


47 


100 


0.74 


[0.46, 


1.20] 


Item 24 


180 


212 


87 


100 


0.84 


[0.42, 


1.68] 


Item 25 


146 


212 


45 


100 


2.70 


[1.66, 


4.41] 


Item 26 


95 


212 


60 


100 


0.54 


[0.33, 


0.88] 


Item 27 


52 


212 


39 


100 


0.51 


[0.31, 


0.85] 



I h- 

0.01 0.1 



10 100 



Figure 5. Comparison of the quality of reporting of included systematic reviews/Meta-analyses based on the affiliations. 

doi:1 0.1 371 /joumal.pone.0085908.g005 



PRISMA Checklist Assessment (Table 3) 

Generally compliance with PRISMA was poor: none of 
included 312 SRs/MAs fulfilled all 27 items of PRISMA. 70.0% 
of PRISMA criteria were complied within the following items: title 
(item 1, 82.4%), rational (item 3, 89.7%), eligibility criteria (item 6, 
88.1%), information sources (item 7, 87.8%), summary measures 
(item 13, 74.4%), synthesis of results (item 14, 85.3%), study 
characteristics (item 18, 78.8%), results of individual studies (item 
20, 84%), summary of evidence (item 24, 85.6%). Total 
compliance with PRISMA items was less than 30.0% in structured 
summary (item 2, 22.4%), objective (item 4, 18.9%), protocol and 
registration (item 5, 2.6%), risk of bias across studies (item 15, 
26.3%), funding (item 27, 28.8%). Detailed information was 
presented in Table 3. 

Subgroup Analysis on the Quality of Reporting 
(Figures 4-8) 

Comparison of the quality of reporting of included SRs/ 
MAs based on the publication time (£2008 vs. >2009). To 

determine whether there had been an improvement in the quality 
of reporting after the PRISMA statement was released. The full 
compliance in each of the PRISMA criteria were compared 



between the period in or before 2008 and the period in or after 
2009 (Figure 4). The results showed that there was an improve- 
ment in the following items after the PRISMA statement was 
released, which was significant difference (P<0.050): structured 
summary (item 2), rational (item 3), objective (item 4), data items 
(item 1 1), risk of bias in individual studies (item 12), study selection 
(item 17), study characteristics (item 18), risk of bias with studies 
(item 19), limitations (item 25). However, other items didn't show 
significant difference or there was no improvement after the 
PRISMA statement was released. 

Comparison of the quality of reporting of included SRs/ 
MAs based on the affiliations (Hospital vs. University). To 
investigate whether there had been a difference in the quality of 
reporting in different affiliations. SRs/MAs published by hospital 
were compared with that published by university for full 
compliance in each of the PRISMA criteria (Figure 5). The 
studies published by hospital had an improvement, and there had 
been a significant difference in limitations (item 25). 

Comparison of the quality of reporting of included SRs/ 
MAs based on the funding sources (Funded vs Non- 
funded). The funded articles were compared with non-funded 
articles for full compliance in all the PRISMA criteria (Figure 6). 
We found that funded articles had an improvement and there was 
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Study or Subgroup 



Fund 
Events Total 



Non-fund 
Events Total 



Odds Ratio 
M-H. Fixed. 95% CI 



Odds Ratio 
M-H. Fixed. 95% CI 




Item 1 


72 


87 


186 


225 


1.01 [0.52, 1.94] 


Item 2 


22 


87 


48 


225 


1.25 [0.70, 2.23] 


Item 3 


82 


87 


198 


225 


2.24 [0.83, 6.01] 


Item 4 


24 


87 


35 


225 


2.07 [1.14, 3.74] 


Item 5 


5 


87 


3 


225 


4.51 [1.05, 19.31] 


Item 6 


79 


87 


196 


225 


1.46 [0.64, 3.33] 


Item 7 


75 


87 


199 


225 


0.82 [0.39, 1.70] 


Item 8 


47 


87 


73 


225 


2.45 [1.48,4.06] 


Item 9 


39 


87 


95 


225 


1.11 [0.68, 1.83] 


Item 10 


59 


87 


134 


225 


1.43 [0.85, 2.41] 


Item 1 1 


38 


87 


110 


225 


0.81 [0.49, 1.33] 


Item 12 


59 


87 


145 


225 


1.16 [0.69, 1.97] 


Item 13 


66 


87 


166 


225 


1.12 [0.63, 1.98] 


Item 14 


75 


87 


191 


225 


1.11 [0.55, 2.26] 


Item 15 


31 


87 


51 


225 


1.89 [1.10, 3.24] 


Item 16 


30 


87 


92 


225 


0.76 [0.45, 1.27] 


Item 17 


47 


87 


94 


225 


1.64 [1.00,2.69] 


Item 18 


69 


87 


177 


225 


1.04 [0.57, 1.91] 


Item 19 


61 


87 


140 


225 


1 .42 [0.84, 2.43] 


Item 20 


73 


87 


178 


225 


1.38 [0.71, 2.65] 


Item 21 


71 


87 


164 


225 


1.65 [0.89, 3.06] 


Item 22 


28 


87 


70 


225 


1.05 [0.62, 1.79] 


Item 23 


32 


87 


99 


225 


0.74 [0.45, 1.23] 


Item 24 


79 


87 


188 


225 


1.94 [0.87, 4.36] 


Item 25 


65 


87 


147 


225 


1.57 [0.90, 2.73] 


Item 26 


51 


87 


104 


225 


1.65 [1.00, 2.72] 


Item 27 


80 


87 


11 


225 


222.34 [83.29, 593.49] 



0.01 0.1 



Figure 6. Comparison of the quality of reporting of included systematic reviews/Meta-analyses based on the funding sources. 

doi:10.1371/journal.pone.0085908.g006 



a significant difference (P<0.050) in objective (item 4), protocol 
and registration (item 5), search (item 8), risk of bias with studies 
(item 15), study selection (item 17), conclusions (item 26), funding 
(item 27). However, there was no statistical difference in other 
items. 

Comparison of the quality of reporting of included SRs/ 
MAs from CSCD vs non CSCD. Chinese Science Citation 
Database (CSCD) has a collection of all core and excellent journals 
in China. By comparing the quality of reporting of SRs/MAs from 
CSCD and from non-CSCD, using the PRISMA criteria 
(Figure 7), we found that there was a statistically significant 
difference (P<0.050) in study selection (item 9), data collection 
process (item 10), summary measures (item 13). However, there 
was not statistical difference in other items. 

Comparison of the quality of reporting of included SRs/ 
MAs from SCI vs non SCI. We included all SRs/MAs 
published in Chinese and English by authors in China. A 
comparison was conducted on the quality of reporting between 
SCI articles and non-SCI articles according to full compliance 
(Figure 8), there was statistical difference in 22 items (81.5%), and 
the quality of SCI articles was better compared with non-SCI 
articles. 



Discussion 

Characteristics of Included Diagnostic SRs or MAs 

The number of diagnostic SRs/MAs written by authors in 
China is increasing annually, 75.3% of SRs/MAs were published 
in China. Nearly 89.0% were published after the year 2007, a 
large part of this probably resulted from the introduction 
QUAD AS checklist [14] in 2007. The diagnostic SRs/MAs 
published by hospitals were 66.3%, indicating that clinicians also 
devoted to evidence production, which would be beneficial to 
evidence transforming clinical practice. Only 1 1.5% research were 
conducted by 1-2 authors, indicating that the result of data 
collection and quality assessment were reliable for most of 
diagnostic SRs/MAs. Most of these articles were published in 
evidence-based medical journals, and they should be an important 
evidence sources for diagnostic tests. The impact factor was higher 
than 5.0 for 12 SRs/MAs. 312 studies involved 15 symptom 
systems, and neoplasms (42.3%) was a hot topic for researchers. 
The imaging technologies and laboratory technologies were main 
diagnostic mode for included studies, which probably due to the 
fact that it is easy to conform the quantitative index and the testing 
threshold in imaging technologies and laboratory technologies. 
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Study or Subgroup 



CSCD 
Events Total 



Non-CSCD 
Events Total 



Odds Ratio 
M-H. Fixed. 95% CI 



Odds Ratio 
M-H. Fixed. 95% CI 



tern 1 


102 


134 


79 


101 


0.89 [0.48, 1.65] 


tern 2 


24 


134 


10 


101 


1.99 [0.90,4.37] 


tern 3 


119 


134 


85 


101 


1.49 [0.70, 3.18] 


tern 4 


7 


134 


5 


101 


1.06 [0.33, 3.44] 


tern 5 


0 


134 


0 


101 


Not estimable 


tern 6 


120 


134 


92 


101 


0.84 [0.35, 2.02] 


tern 7 


114 


134 


84 


101 


1.15 [0.57, 2.34] 


tern 8 


58 


134 


32 


101 


1.65 [0.96, 2.83] 


tern 9 


57 


134 


26 


101 


2.14 [1.22, 3.75] 


tern 10 


87 


134 


42 


101 


2.60 [1.53,4.42] 


tern 11 


54 


134 


33 


101 


1.39 [0.81, 2.39] 


tern 12 


82 


134 


51 


101 


1.55 [0.92, 2.61] 


tern 13 


81 


134 


80 


101 


0.40 [0.22, 0.73] 


tern 14 


109 


134 


80 


101 


1.14 [0.60, 2.19] 


tern 15 


23 


134 


19 


101 


0.89 [0.46, 1.75] 


tern 16 


54 


134 


32 


101 


1.46 [0.85, 2.51] 


tern 17 


50 


134 


32 


101 


1.28 [0.74, 2.22] 


tern 18 


99 


134 


73 


101 


1.08 [0.61, 1.94] 


tern 19 


78 


134 


53 


101 


1.26 [0.75, 2.12] 


tern 20 


106 


134 


75 


101 


1.31 [0.71, 2.42] 


tern 21 


98 


134 


69 


101 


1.26 [0.72, 2.23] 


tern 22 


31 


134 


26 


101 


0.87 [0.48, 1.58] 


tern 23 


56 


134 


37 


101 


1.24 [0.73, 2.11] 


tern 24 


111 


134 


83 


101 


1.05 [0.53, 2.06] 


tern 25 


89 


134 


55 


101 


1.65 [0.97, 2.81] 


tern 26 


55 


134 


42 


101 


0.98 [0.58, 1.65] 


tern 27 


35 


134 


28 


101 


0.92 [0.52, 1.65] 



0.01 



— I— 

0.1 



-+- 



— I 1 

10 100 



Figure 7. Comparison of the quality of reporting of included systematic reviews/Meta-analyses from CSCD vs non CSCD. 

doi:1 0.1 371 /journal.pone.0085908.g007 



The Quality of Reporting of Included Diagnostic SRs/MAs 

Our PRISMA results showed that the quality of reporting still 
needs further improvement, the main problems were identified in 
following four areas: The compliance with PRISMA was not good 
in structured summary and objectives. The reporting flaws were 
found in most of (50.0%) included reviews in methods section: 
protocol and registration, search, study selection, data items, risk 
of bias across studies, and additional analyses including subgroup 
analysis and sensitivity analysis). These flaws would affect the 
quality of diagnostic SRs/MAs. The reporting rate of study 
selection, risk of bias across studies and additional analyses was less 
than 50.0% in results section. Even though some of SRs/MAs 
reported risk of bias across studies and additional analyses in 
methods section, the analytic results were not presented in results 
section. Similarly, some of SRs/MAs reported risk of bias across 
studies and additional analyses in results section, but the analytic 
results were not reported in methods section. It would affect the 
integrity and accuracy of researches. The reporting of conclusions 
was incomplete, most of studies only provided a general 
interpretation of the results, however, the reporting of implications 
for future research was poorly provided. 



Comparisons of the Quality of Reporting of SRs/MAs 
Published by Authors in China and Authors Abroad 

Willis BH et al have investigated the quality of reporting of 
meta-analyses in diagnostic tests published by international 
authors prior to December 31 st , 2008 [1]. We compared the total 
compliance (Yes) in each item of PRISMA by t-test in regarding to 
SRs/MAs published by authors in China and authors abroad 
(Text S2). The results showed that the quality of reporting of 
diagnostic SRs/MAs published by authors in China had more 
serious reporting flaws in the following items when compared with 
SRs/MAs published by authors abroad, and the difference was 
statistically significant (the compliance for PRISMA was less than 
50.0%): structured summary (22.4% vs. 34.8%, P = 0.002), 
objectives (18.9% vs. 66%, P<0.001), risk of bias across studies 
(methods section, 26.3% vs. 64.4%, P<0.001), risk of bias across 
studies (results section, 34.4% vs. 57.6%, P<0.001), funding 
(28.9% vs. 48.3%, P<0.001). It is worth mentioning that the 
reporting quality of diagnostic SRs/MAs published by authors in 
China was higher statistically in following items when compared 
with SRs/MAs published by authors abroad: search (methods 
section, 38.5% vs. 25.0%, P<0.001), risk of bias in individual 
studies (methods section, 65.4% vs. 43.6%, P<0.001), study 
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Study or Subgroup 



SCI Non-SCI 
Events Total Events Total 



Odds Ratio 
M-H. Fixed. 95% CI 



Odds Ratio 
M-H. Fixed. 95% CI 



Item 1 


74 


74 


184 


238 


44.01 [2.68, 721.95] 


Item 2 


35 


74 


35 


238 


5.21 [2.91, 9.30] 


Item 3 


74 


74 


206 


238 


23.45 [1.42, 387.79] 


Item 4 


45 


74 


14 


238 


24.83 [12.16, 50.69] 


Item 5 


7 


74 


1 


238 


24.76 [2.99, 204.80] 


Item 6 


62 


74 


214 


238 


0.58 [0.27, 1.22] 


Item 7 


74 


74 


201 


238 


27.73 [1.68,457.31] 


Item 8 


28 


74 


92 


238 


0.97 [0.56, 1.65] 


Item 9 


48 


74 


86 


238 


3.26 [1.89, 5.63] 


Item 10 


61 


74 


132 


238 


3.77 [1.97, 7.23] 


Item 1 1 


58 


74 


90 


238 


5.96 [3.23, 11.00] 


Item 12 


68 


74 


136 


238 


8.50 [3.55, 20.36] 


Item 13 


69 


74 


163 


238 


6.35 [2.46, 16.39] 


Item 14 


74 


74 


192 


238 


35.99 [2.19, 591.57] 


Item 15 


38 


74 


44 


238 


4.65 [2.66, 8.16] 


Item 16 


34 


74 


88 


238 


1.45 [0.85,2.46] 


Item 17 


57 


74 


84 


238 


6.15 [3.36, 11.24] 


Item 18 


71 


74 


175 


238 


8.52 [2.59, 28.02] 


Item 19 


67 


74 


134 


238 


7.43 [3.27, 16.86] 


Item 20 


67 


74 


184 


238 


2.81 [1.22, 6.48] 


Item 21 


65 


74 


170 


238 


2.89 [1.36, 6.13] 


Item 22 


39 


74 


59 


238 


3.38 [1.96, 5.82] 


Item 23 


36 


74 


95 


238 


1.43 [0.84, 2.41] 


Item 24 


70 


74 


197 


238 


3.64 [1.26, 10.54] 


Item 25 


66 


74 


146 


238 


5.20 [2.39, 11.33] 


Item 26 


56 


74 


99 


238 


4.37 [2.42, 7.88] 


Item 27 


26 


74 


65 


238 


1.44 [0.83,2.51] 



-I— ► 



0.01 0.1 



—\ 1 

10 100 



Figure 8. Comparison of the quality of reporting of included systematic reviews/Meta-analyses from SCI vs non SCI. 

doi:1 0.1 371 /journal.pone.0085908.g008 



characteristics (results section, 78.9% vs. 55.9%, P<0.001), results 
of individual studies (results section, 85.0% vs. 68.6%, P = 0.020). 

Limitations 

Although both Chinese and English databases were searched, 
and study selection, data extraction and quality assessment were 
conducted independently by two reviewers, there are still some 
limitations to our study due to the different level of understanding 
PRISMA statement between different researchers. Second, our 
study only included diagnostic SRs/MAs, however, the SRs/MAs 
on pathogenesis, prognosis, genetic polymorphism, and interven- 
tion were not included. Third, in our search, we searched the term 
"systematic review" and "meta-analysis" in titles, abstracts or 
keywords. Some potentially eligible systematic reviews may, 
however, not use these terms in their titles or abstracts. PRISMA 
statement is generic checklist aimed at improving the reporting of 
all types of SRs and does not contain some of the more specific 
nuances of diagnostic test reviews. However, most of items of 
PRISMA are applicative regarding to diagnostic SRs/MAs, 
especially eligibility criteria, search, risk of bias with studies, risk 
of bias across studies, and additional analysis etc. The quality of 
reporting for these items was presented in Table 2. Finally, the 
included reviews evaluated a range of diagnostic tests involving 1 5 
symptom systems and different diagnostic technologies. It is 



unlikely that the quality of reporting will be completely 
independent of variation in these factors, therefore, the effect of 
heterogeneity on the results needs to be considered. 

Conclusion 

We have investigated the quality of reporting of 312 SRs/MAs 
related to diagnostic tests published by authors in China. The 
results demonstrate that the number of diagnostic SRs/MAs is 
increasing annually. The quality of reporting has measurably 
improved over the previous years. Unfortunately, there are still 
many flaws in the areas including protocol and registration, 
search, risk of bias across studies, additional analyses. We propose 
following strategies for future research: Since a protocol can pre- 
specify the objectives and methods of the systematic review [1 1], it 
is important to provide registration information including the 
registration number. There is great demand to establish a registry 
platform for SRs/MAs in China. The instructions to authors 
should include related reporting items, and the submission of 
PRISMA checklist. The researchers should update timely 
knowledge on SRs/MAs. The reviewers writing SRs/MAs, 
especially Chinese reviewers, should pay more attention to area 
such as search, risk of bias across studies, additional analyses. 
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